Skip to content

Improve Counterexamples#169

Open
rcosta358 wants to merge 9 commits intomainfrom
improve-counterexamples
Open

Improve Counterexamples#169
rcosta358 wants to merge 9 commits intomainfrom
improve-counterexamples

Conversation

@rcosta358
Copy link
Collaborator

@rcosta358 rcosta358 commented Mar 4, 2026

This PR improves the counterexamples by filtering out assignments that are already present in the context, since these don't add any new information to the verification state filtering out assignments of variables that don't appear in the error message.

Examples


Before

image

After

image

Before

image

After

image

@rcosta358 rcosta358 self-assigned this Mar 4, 2026
@CatarinaGamboa
Copy link
Collaborator

I'm not sure I get what the goal is here. Its because we are already showing the most simplified version in the refinement error that we don't need the other values?

I'm especially confused about the 2nd example, its very weird that we show #a_2 == X and not #a_1. Should we also be removing #a_2 since it is nowhere in the simplified error message?

Also, it raises a question, should we just present the simplified error version in the console version? in the ide we can expand it but here could be helpful to also present the full-on error message to match the counterexample.

(unrelated but these errors will be much better when we apply #162 )

Copy link
Collaborator

@CatarinaGamboa CatarinaGamboa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to understand the objective better ^^

@rcosta358
Copy link
Collaborator Author

This is not really about filtering out assignments that are not present in the simplified expression of the error message, but about filtering out the assignments that are already present in the context. This change resulted from a conversation with @alcides, where we agreed that if the context already includes a refinement shown in the counterexample, we consider it redundant and remove it to simplify the counterexample.

@rcosta358
Copy link
Collaborator Author

rcosta358 commented Mar 5, 2026

Regarding showing unsimplified expressions in the command-line version, maybe we could add a command argument like --no-simplification to do so, but I think always showing both versions would be too verbose.

@alcides
Copy link
Collaborator

alcides commented Mar 5, 2026

The main point is:

void f(@Refined("x > 0") int x) {
   int a = 3;
   f(x - a);
}

In this case, the counter example showing that a == 3 is not very useful, because it was assigned before (there may be thousands of variables in the body, including those from anf). We care about the x, because its refinement is not an equality, so a proper witness matters.

Does this make sense? or am I missing some other case, where showing other variables (we know about) is relevant?

In counterexample, check if assignment is already present in all variable refinements, not just instance variable refinements.
@CatarinaGamboa
Copy link
Collaborator

We already do a filtering of the context refinements sent to the smt solver for each check, so we should only get the relevant variables in the counter example right

@alcides
Copy link
Collaborator

alcides commented Mar 6, 2026

It’s different. We are talking about formulas that are relevant for type-checking but do not add anything new to what the programmer already knows.

if they know a = 3 but b > 0, the counter example of b will be relevant, but the counter example of a will be 3, as expected from the source code.

@rcosta358
Copy link
Collaborator Author

Yes, if we send the refinement x = 1 to the SMT solver, x = 1 appearing in the counterexample is just not useful.

@CatarinaGamboa
Copy link
Collaborator

Ok but still you are assuming that the developer remembers that x = 1, and it's not too far away in the program from the place the error shows up.
I think we could have 2 options for both the refinement found and the counter example: 1) simplified version, and 2) with the complete information. In the IDE we can hide it and reveal after upon need, but in the text message we can start by having the simplified version and then a bar with the complete info or something

@rcosta358 rcosta358 marked this pull request as draft March 12, 2026 14:19
@rcosta358 rcosta358 marked this pull request as ready for review March 12, 2026 16:34
@rcosta358
Copy link
Collaborator Author

@alcides because of the concerns that @CatarinaGamboa raised, instead of filtering out assignments in the counterexample that are already present in the context, we now filter out assignments of variables that don't appear in the error message, which actually shows better results than the previous approach.

Copy link
Collaborator

@CatarinaGamboa CatarinaGamboa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach is much better imo!
For the implementation, let's use expression traversals as much as possible instead of string splits and substrings - since this is a compiler, we already have the mechanisms in place to walk the AST, and using them should keep things more robust and maintainable

.reduce((a, b) -> a + " && " + b).orElse("");
// filter out assignments of variables that do not appear in the found value
String foundValue = found.getValue().toString();
List<String> foundTokens = Arrays.asList(foundValue.split("[^a-zA-Z0-9_#]+"));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to treat this as a string right? found.getValue() gives you an Expression, you can traverse the expression tree and see the equals operations and the names

return "Counterexample: " + getCounterExampleString();
String counterexampleString = getCounterExampleString();
if (counterexampleString == null)
return "";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

*
* @return list of all variables
*/
public List<RefinedVariable> getAllVariables() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was duplicated?

Copy link
Collaborator Author

@rcosta358 rcosta358 Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's just not being used anymore

@rcosta358
Copy link
Collaborator Author

You're right, I overlooked that detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants